Integrating Random Testing with Constraints for Improved Efficiency and Diversity

Random testing can be fully automated, eliminates subjectiveness in constructing test cases, and increases the diversity of test data. However, randomly generated tests may not satisfy program’s assumptions (e.g., method preconditions). While constraint solving can satisfy such assumptions, it does not necessarily generate diverse tests and is hard to apply to large programs. We blend these techniques by extending random testing with constraint solving, improving the efficiency of generating valid test data while preserving diversity. For domains such as objects, we generate input values randomly; however, for values of finite domains such as integers, we represent test data generation as a constraint satisfaction problem by solving constraints extracted from the precondition of the method under test. We also increased the diversity of constraint-based solutions by incorporating randomness into the solver’s enumeration process. In our experimental evaluation we observed an average improvement of 80 times without decreasing test case diversity, measured in terms of the time needed to generate a given number of test cases.